Searchable index

ABSTRACT

Systems and techniques are disclosed for generating entries for a searchable index based on rules generated by one or more machine-learned models. The index entries can include one or more tokens correlated with an outcome and an outcome probability. A subset of tokens can be identified based on the characteristics of an event. The index may be searched for outcomes and their respective probabilities that correspond to tokens that are similar to or match the subset of tokens based on the event.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.14/268,049, filed May 2, 2014, the contents of which are incorporated byreference herein.

BACKGROUND

Recommendation systems can have separate information retrieval andmachine-learned ranking stages. The information retrieval stage selectsdocuments (videos, advertisements, music, text documents, etc.) from acorpus based on various signals and the machine-learned system ranks theoutput of the information retrieval system. For example, when a userenters a query “cat”, a contextual information retrieval system mayselect a set of candidate advertisements that contain the word “cat”from all available advertisements. This set of candidate advertisementscan then be ranked based on a machine-learned model that has beentrained to predict the likelihood of an advertisement being clickedthrough by a user based on various features, such as the type of user,the location of the user, the time of day at which the query was made,etc. An information retrieval tool is computationally efficient, but canonly produce a rough estimate of which items are best recommended to auser. A machine-learned model can produce more accurate recommendations,but is often more computationally intensive than an informationretrieval tool. Further, because the information retrieval tool is lessaccurate, it can exclude certain candidates from consideration using themachine-learned model that would otherwise be highly ranked.

BRIEF SUMMARY

According to implementations of the disclosed subject matter, aplurality of rules from a machine learning model may be generated by acomputerized machine learning system and received, each rule containingan outcome, one or more features, and an outcome probability predictedby the machine learning model for the one or more features and theoutcome. An entry for a token-based index may be generated for each ofthe plurality of rules and may contain one or more tokens based on theone or more features of the rule, the outcome of the rule, and theoutcome probability of the rule. A query may be received and a subset oftokens that correspond to the query may be identified. The token-basedindex may be searched to obtain a plurality of outcome probabilitiesbased on the subset of tokens. An outcome may be selected based on theplurality of outcome probabilities and may be provided to a user. Asubset of selected outcomes may be ranked. For example, a hundred videos(outcomes) may be ranked by probability and the top twenty may be shown,from highest to lowest probability, to the user.

Systems and techniques according to the present disclosure may generateor modify one or more indices based on rules and results of a model thatis the product of a machine learning system. The generated or modifiedindices may be used to provide results based on a search technique.Additional characteristics, advantages, and implementations of thedisclosed subject matter may be set forth or apparent from considerationof the following detailed description, drawings, and claims. Moreover,it is to be understood that both the foregoing summary and the followingdetailed description include examples and are intended to providefurther explanation without limiting the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the disclosed subject matter, are incorporated in andconstitute a part of this specification. The drawings also illustrateimplementations of the disclosed subject matter and together with thedetailed description serve to explain the principles of implementationsof the disclosed subject matter. No attempt is made to show structuraldetails in more detail than may be necessary for a fundamentalunderstanding of the disclosed subject matter and various ways in whichit may be practiced.

FIG. 1 shows an example illustration for providing outcomes/outcomeprobabilities via a token-based index, according to an implementation ofthe disclosed subject matter.

FIG. 2a shows an example process of generating a token-based index,according to an implementation of the disclosed subject matter.

FIG. 2b shows an example process of searching a token-based index foroutcomes based on outcome probabilities, according to an implementationof the disclosed subject matter.

FIG. 3 shows a computer according to an implementation of the disclosedsubject matter.

FIG. 4 shows a network configuration according to an implementation ofthe disclosed subject matter.

DETAILED DESCRIPTION

According to implementations of the disclosed subject matter, asearchable index may be generated based on rules derived from amachine-learned model. In this way, the superior intelligence and logicof a machine-learned model can be embodied in an easily-searchableindex. Standard information retrieval tools can be used to efficientlyretrieve data using the index. This can compare favorably to a systemthat has separate information retrieval and machine-learned rankingstages by eliminating the loss of candidates during the informationretrieval stage that would otherwise be highly scored by themachine-learned model. Overall, more candidates can be more accuratelyranked in accordance with an implementation of the disclosed subjectmatter.

The techniques disclosed herein may be applied to machine-learned modelsgenerated using supervised learning data such as labeled examples. Thelabeled examples may be input into a machine learning system and theoutput from the machine learning system may be a machine-learned modelthat contains weights that are generated in response to the labeleddata. The labeled examples may contain both an outcome and propertiesassociated with a specific instance. It will be understood that theweights and outcomes may be unitless numerical values, percentages,counts of occurrences, or other quantification. A machine learningsystem may receive labeled data (e.g., labeled examples) to develop amachine-learned model that contains weights that are generated inresponse to the supervised labeled data. One or more rules may becreated by a machine-learned model.

A rule can include an outcome, a set of features and a probability. Forexample, the rule:

(keyword:car, video:carmaker_1)→0.03

indicates that when a user submits a search keyword “car” at a searchengine (a feature) and the user is presented with an video aboutcarmaker_1 (the outcome), there is a 3% probability that the user willselect the video to view it (the probability).

The entries in a searchable index can include documents and searchabletokens. A token in a machine-learned token-based index can be referredto as an indexed token. An indexed token may include a keyword or maynot include a keyword. For example, an index can include one token thatincludes the keyword “car” as well as other tokens that do not includekeywords but that relate to other features such as location, languageand browser settings. In general, a feature can include any informationknown about a user, such as a query text submitted by the user, abrowser configuration set by the user, etc. Also, a feature can begeneral state information such as the time of day, a geographiclocation, etc. For example, a webpage can be indexed to produce an entryas follows:

web_page_1: [text:boxcar, 4.0], [image:train, 2.0]

This indicates that a page of a website (“web_page_1”, a document)includes the text string “boxcar” four times and two images of trains.

In accordance with an implementation of the disclosed subject matter, astandard rule-based machine-learned model can be decomposed into a setof documents and tokens with weights. For example, the following rulesindicate the likelihood that a user who enters the search keyword “car”into a search engine will select a video about a particular carmaker:

(keyword:car, video:carmaker_1)→0.03

(keyword:car, video:carmaker_2)→0.05

These rules can be transformed into entries that include a set ofsearchable tokens corresponding to each video, such as:

carmaker_1: [keyword:car, 0.03]

carmaker_2: [keyword:car, 0.05]

An entry can include an outcome (such as “carmaker_1”), tokens (such askeyword:car) and a weight (such as 3%). The tokens can be based on theoccurrence of features in a machine-learned rule. A weight cancorrespond to the probability that the outcome will occur based on theoccurrence of certain features, represented here as tokens.

Since the data of the implementation has the same structure as astandard web search information retrieval problem, standard web searchand other information retrieval techniques can be used, such as invertedindices and posting lists. An implementation thereby can eliminate theneed for a separate information retrieval step and can effectively scoreall of the outcomes using the machine-learned model. In this way, searchresults can be provided that may be more appropriate to a user than atypical indexed search, for example that may be based on attributes ofthe user, context of the search, etc.

In an implementation, features can be associated with a given user. Forexample, the following table shows the presence (1) or absence (0) offeatures (query, user location (Europe, America) and whether the userhas a high bandwidth connection) for Users A-D:

High Car Europe America Bandwidth User A 0 0 1 1 User B 1 0 1 0 User C 01 0 1 User D 1 1 0 1

A machine-learned model can include weights that can indicate therelative contributions of various features to the likelihood of anoutcome. For example, the table below shows the relative contributionsof the features in the table above to the likelihood that a user willselect for viewing a particular video A-D:

High Car Europe America Bandwidth Video A 0.1 0.4 0.8 0.5 Video B 0.30.9 0.7 0.8 Video C 0.5 0.1 0.2 0.1 Video D 0.2 0.6 0.2 0.8

The presence or absence of a feature for a given user can be used withthe weight of the feature for an outcome to determine the likelihood ofthat outcome for that user. For example, the presence of keyword:car andAmerica for User B combined with the weights for those features forVideo C (0.5 and 0.2, respectively) can be used to predict a probabilitythat User B will select Video C for viewing. In similar fashion, thelikelihood that each user will choose to view each different video canbe calculated. It will be understood that the weights may be unitlessnumerical values, percentages, counts of occurrences, or otherquantification.

Rules can be generated based on the machine-learned model. A rule cancorrelate at least one feature with a probability of occurrence of agiven outcome. A number of rules can be generated based on theoccurrences of various permutations of features.

In accordance with an implementation, a set of tokens can be generatedbased on the features in a rule. Examples of such tokens can include[keyword:car], [location:Europe] and [bandwidth:high]. A set of tokenscorresponding to a set of features in a rule can be correlated with aprobability of a given outcome and indexed. For example, the tokens[keyword:car], [location:Europe] and [bandwidth:high] can be correlatedwith a 4% probability that Video C will be selected to be viewed:

Video C: [keyword:car, location:Europe, bandwidth:high, 0.04]

This can be stored as a searchable index along with other such resultsderived from other rules. The compiled index can be easily searchedusing standard information retrieval tools. The generated index isreferred to as a token-based index. The index is based on tokens thatare not limited to keywords. Optionally, one or more ranking techniquescan thereafter be employed to further refine the search results inresponse to a query.

The index can be used to consider only those tokens corresponding tofeatures for a given user. For example, a search of the index canretrieve all videos for which probabilities have been calculated for auser whose last viewed video was about cats and whose location isAmerica. The resulting videos can be ranked by probability from most toleast likely to be selected next for viewing when presented assubsequent viewing options to the user. Descriptions (e.g., thumbnails)for the top-ranked videos in the list can be recommended to the user.

An outcome may be one for which the machine learning model predicted anoutcome value. The value may be a weight, likelihood, or probability. Byway of example only, we refer to probability herein. The outcome may beany applicable result such as a regression-based prediction, a ranking,a conversion probability, a click through probability, a durationprediction (e.g., how long a user is likely to view/interact withcontent associated with the outcome), or the like. As discussed in theprevious example, a user viewing video Y and a user viewing the video Zare examples of outcomes. Other examples of outcomes may be selecting alink, selecting a promotion, opening an account, purchasing a product orservice, the duration for which a user views a content (e.g., a video,an image, a text, etc.), repeat access (e.g., how likely a user is torevisit content), or the like.

An outcome probability may be represented in any useful form, includinginteger representations, Boolean categorization, normalization (e.g.,the probability value converted into a normalized probability,conversion rate, percentage, etc.). It will be understood that anoutcome probability may be any applicable prediction such as apercentage, ratio, or the like and/or may correspond to a prediction ofamount spent (e.g., dollars spent), amount of exposure time (e.g., videominutes watched), or the like. As disclosed herein, the outcomeprobability may be derived from the prediction made by the machinelearning model. As an example of an outcome probability, a searchableindex may contain multiple entries, each associated with an outcome.Each entry may correspond to an outcome probability that predicts thelikelihood of a user selecting the content associated with the outcome.Accordingly, the outcome probability may represent the percentage chanceof a user selecting content associated with the outcome.

Notably, according to implementations of the disclosed subject matter,as shown in FIG. 1, a machine-learned model 110 may be trained based ontraining examples and features. A searchable token-based index 120 maybe generated based on rules generated by the machine-learned model. Thesearchable token-based index 120 may be organized such that tokensassociated with a query 130 may be matched with the tokens in thesearchable token-based index 120 and, using a search algorithm, anoutcome/outcomes 140 may be selected based on one or more outcomeprobabilities.

According to an implementation of the disclosed subject matter, amachine learning system may be used to generate and/or update modelsthat are used to make predictions and/or provide rankings. An instanceof a machine learning model-based prediction may contain an outcome, oneor more features, and a prediction value. As an example of a predictionmade via a machine learning model, multiple features for a given eventmay be provided to the machine learning model and, based on the presenceof the features, the machine learning model may output a probability orprediction. As a more specific example, a machine learning model thatpredicts whether a user will view video Y (an example of an outcome) maybe provided with features that the user located in the United States,has viewed a music video X in the past, and has set her default languageas English (examples of features). The machine learning model maycontain weights for each of the features (i.e., 0.5 for being located inthe United States, 0.9 for having viewed music video X, and 0.3 forsetting the default language as English). It will be understood that themachine learning model may contain weights for other features (e.g. theuse is located in Canada) however, as those features are not present inthis example prediction, their weights may not contribute to theprediction. Alternatively, the lack of presence of a particular featuremay be important in predicting an outcome and may be taken into account.For example, the machine learning model may provide a weight for whetherthe user will view music video Y based on the absence of a feature(e.g., the user is not using a mobile device). The prediction value maybe normalized to represent a percentage or probability in any applicablemanner. Here the instance may contain the outcome: “whether the userwill view video Y”, the features: “located in the United States”,“viewed video X”, and “default language English”, and the prediction:“0.9” (normalized).

A feature may be any applicable characteristic associated with aninstance and may be based on a user (e.g., user demographic, userhistory, user associations such as user accounts or preferences), adevice (e.g., a user device type, device features, device capabilities,device configuration etc.), a current event, or the like. Examples offeatures can include a user location, a user language preference, a viewhistory, a user account, a user subscription, a device model type, adevice screen resolution, a device operating system, a holidaydesignation, a sporting event occurrence, or the like. Additionally, asdisclosed herein, a feature may be a search input (such as a textkeyword, an image, an audio file, etc.) received from a user. It will beunderstood that the outcome, features, and/or the prediction may berepresented in any useful form such as integer representations, viaBoolean categorization, normalized (e.g., the probability valueconverted into a normalized probability, conversion rate, percentage,etc.). According to an implementation of the disclosed subject matter,the machine learning model may be trained using prior selections of onemore users (outcomes), as disclosed herein. Notably, the prediction madeby the machine learning model may be based on the successful selectionor lack of selection of an outcome such that the predicted outcomeprobability may increase based on selected candidate results and maydecrease based on unselected outcomes.

According to an implementation of the disclosed subject matter, as shownin FIG. 2a , at step 210, multiple rules based on a machine learningmodel may be received. An instance of a machine learning model-basedrule may contain an outcome, one or more features, and an outcomeprobability. As an example of a prediction made via a machine learningmodel, multiple features for a given event may be provided to themachine learning model and, based on the presence of the features, themachine learning model may output a probability or prediction. As a morespecific example, a machine learning model that predicts whether a userwill view video Y (an outcome) may be provided with data that the userlocated in the United States, has viewed a music video X in the past,and has set her default language as English (features). The machinelearning model may prescribe weights for each of the features, e.g., 0.5for being located in the United States, 0.9 for having viewed musicvideo X, and 0.3 for setting the default language as English.Accordingly, the machine learning model may predict that the user willview music video Y with a weight of 1.7 based on the features associatedwith the rule. The probability value may be normalized to represent apercentage or probability in any applicable manner. Here the instancemay contain the outcome: “whether the user will view video Y”, thefeatures: “located in the United States”, “viewed video X”, and “defaultlanguage English”, and the prediction: “0.9” (normalized). It will beunderstood that the outcome, features, and/or the probability may berepresented in any applicable manner such as hash values, integerrepresentations, Boolean categorization, normalization (e.g., theprobability value converted into a normalized probability, conversionrate, percentage, etc.). As an example, an outcome for “Selecting videoX” may be represented with a hash value “e0d123e5f316”.

At step 220, an instance of a rule based on a machine learning model maybe converted into an entry in a searchable feature based index. Theentry in the searchable feature based index may contain an outcomeassociated with one or more tokens and an outcome probability. A tokenmay be based on a feature contained within a rule. Continuing theprevious example, the model may predict a probability of 0.9 for theoutcome “the user will view video Y” based on various features. Thetoken-based index may include the tokens location:US, viewed_video:X,and “language:English” correlated with the outcome “the user will viewvideo Y” and a probability of 0.9. The token-based index may correlatethe same tokens to other outcomes, each with its own probability. Forexample, the same tokens may be correlated to the outcome “the user willview video Z” with a probability of 0.8.

A searchable token-based index may be an inverted index or a postinglist such that it is an index data structure that is configured to storea mapping from content (e.g., words, numbers, values, etc.) to locationsin a database file, to documents, or a set of documents. The searchabletoken-based index may allow fast full text searches and may be adatabase file itself rather than its index.

At step 230, in FIG. 2b , a query may be received. A query may begenerated based on an event caused by a human user, a computer, adatabase, software, an application, a server, or the like. As usedherein, the term query may include any input that can be used to searchthe index to obtain a probabilities of one or more outcomes based on theoccurrence of one or more events. For example, when a user selects agiven video, the characteristics of the selection (e.g., identity of thevideo, topic of the video, location of the user, etc.) can be used asthe basis of a query to search the index for outcomes and theirrespective probabilities that the user will select other videos to watchnext. The results of the query can predict, for example, that the userwill select Video B with a probability of 0.2, Video C with aprobability of 0.1 and Video D with a probability of 0.4.

In an implementation, a query can be formulated based on a subset oftokens that may be identified, step 240, e.g., based on an event. Forexample, a keyword search for “car” may have been submitted by a user inCanada at 5:07 PM ET with the language setting of the user's browser setto “French.” The subset of tokens that may be identified can includekeyword:car, location:Canada, time:5:07 PMET and language:French. Thesetokens can be used to search the index for outcomes and probabilitiescorrelated with the same or similar tokens in the index, step 250. Forexample, the tokens associated with a query may be:

keyword: car location: USA bandwith: highThese tokens may correspond, for example, to the following entries inthe index, which can efficiently be retrieved using standard indexsearch techniques:

Video ID Tokens associated with entry % Video A keyword: car location:USA bandwith: high 45 Video C keyword: car location: USA bandwith: high24 Video F keyword: car location: USA bandwith: high 76 Video R keyword:car location: USA bandwith: high 55

One or more outcomes (above, videos) may be selected from the aboveresults, e.g., based on their respective outcome probabilities. As shownabove, the outcome Video F has the highest probability of being selectedfor viewing next. Accordingly, a link to Video F may be provided to auser. Similarly the next highest-ranked (most probable) videos (Video Rand Video A) may also be presented.

Implementations of the presently disclosed subject matter may beimplemented in and used with a variety of component and networkarchitectures. FIG. 3 is an example computer system 20 suitable forimplementing embodiments of the presently disclosed subject matter. Thecomputer 20 includes a bus 21 which interconnects major components ofthe computer 20, such as one or more processors 24, memory 27 such asRAM, ROM, flash RAM, or the like, an input/output controller 28, andfixed storage 23 such as a hard drive, flash storage, SAN device, or thelike. It will be understood that other components may or may not beincluded, such as a user display such as a display screen via a displayadapter, user input interfaces such as controllers and associated userinput devices such as a keyboard, mouse, touchscreen, or the like, andother components known in the art to use in or in conjunction withgeneral-purpose computing systems.

The bus 21 allows data communication between the central processor 24and the memory 27. The RAM is generally the main memory into which theoperating system and application programs are loaded. The ROM or flashmemory can contain, among other code, the Basic Input-Output system(BIOS) that controls basic hardware operation such as the interactionwith peripheral components. Applications resident with the computer 20are generally stored on and accessed via a computer readable medium,such as the fixed storage 23 and/or the memory 27, an optical drive,external storage mechanism, or the like.

Each component shown may be integral with the computer 20 or may beseparate and accessed through other interfaces. Other interfaces, suchas a network interface 29, may provide a connection to remote systemsand devices via a telephone link, wired or wireless local- or wide-areanetwork connection, proprietary network connections, or the like. Forexample, the network interface 29 may allow the computer to communicatewith other computers via one or more local, wide-area, or othernetworks, as shown in FIG. 4.

Many other devices or components (not shown) may be connected in asimilar manner, such as document scanners, digital cameras, auxiliary,supplemental, or backup systems, or the like. Conversely, all of thecomponents shown in FIG. 3 need not be present to practice the presentdisclosure. The components can be interconnected in different ways fromthat shown. The operation of a computer such as that shown in FIG. 3 isreadily known in the art and is not discussed in detail in thisapplication. Code to implement the present disclosure can be stored incomputer-readable storage media such as one or more of the memory 27,fixed storage 23, remote storage locations, or any other storagemechanism known in the art.

FIG. 4 shows an example arrangement according to an embodiment of thedisclosed subject matter. One or more clients 10, 11, such as localcomputers, smart phones, tablet computing devices, remote services, andthe like may connect to other devices via one or more networks 7. Thenetwork may be a local network, wide-area network, the Internet, or anyother suitable communication network or networks, and may be implementedon any suitable platform including wired and/or wireless networks. Theclients 10, 11 may communicate with one or more computer systems, suchas processing units 14, databases 15, and user interface systems 13. Insome cases, clients 10, 11 may communicate with a user interface system13, which may provide access to one or more other systems such as adatabase 15, a processing unit 14, or the like. For example, the userinterface 13 may be a user-accessible web page that provides data fromone or more other computer systems. The user interface 13 may providedifferent interfaces to different clients, such as where ahuman-readable web page is provided to web browser clients 10, and acomputer-readable API or other interface is provided to remote serviceclients 11. The user interface 13, database 15, and processing units 14may be part of an integral system, or may include multiple computersystems communicating via a private network, the Internet, or any othersuitable network. Processing units 14 may be, for example, part of adistributed system such as a cloud-based computing system, searchengine, content delivery system, or the like, which may also include orcommunicate with a database 15 and/or user interface 13. In somearrangements, an analysis system 5 may provide back-end processing, suchas where stored or acquired data is pre-processed by the analysis system5 before delivery to the processing unit 14, database 15, and/or userinterface 13. For example, a machine learning system 5 may providevarious prediction models, data analysis, or the like to one or moreother systems 13, 14, 15.

In situations in which the implementations of the disclosed subjectmatter collect personal information about users, or may make use ofpersonal information, the users may be provided with an opportunity tocontrol whether programs or features collect user information (e.g., auser's performance score, a user's work product, a user's providedinput, a user's geographic location, and any other similar dataassociated with a user), or to control whether and/or how to receiveinstructional course content from the instructional course provider thatmay be more relevant to the user. In addition, certain data may betreated in one or more ways before it is stored or used, so thatpersonally identifiable information is removed. For example, a user'sidentity may be treated so that no personally identifiable informationcan be determined for the user, or a user's geographic locationassociated with an instructional course may be generalized wherelocation information is obtained (such as to a city, ZIP code, or statelevel), so that a particular location of a user cannot be determined.Thus, the user may have control over how information is collected aboutthe user and used by an instructional course provider.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific implementations. However, theillustrative discussions above are not intended to be exhaustive or tolimit implementations of the disclosed subject matter to the preciseforms disclosed. Many modifications and variations are possible in viewof the above teachings. The implementations were chosen and described inorder to explain the principles of implementations of the disclosedsubject matter and their practical applications, to thereby enableothers skilled in the art to utilize those implementations as well asvarious implementations with various modifications as may be suited tothe particular use contemplated.

The invention claimed is:
 1. A computer-implemented method, comprising:before receiving, at a search system, a subsequent query that isassociated with at least one or more features: generating, using apredictive model and by the search system, data indicating a predictedrelevance of a particular item of content to a query that is associatedwith the one or more features, wherein the predicted relevance isgenerated by the predictive model based at least on the one or morefeatures; and storing, by the search system and in a record entry of adatabase for a search index, data that associates the particular item ofcontent and the one or more features with the data indicating thepredicted relevance generated by the predictive model; receiving, at thesearch system, the subsequent query that is associated with at least theone or more features; in response to receiving the subsequent query:accessing, by the search system, the record entry of the database forthe search index to identify the particular item and the predictedrelevance that was previously generated by the predictive model based atleast on the one or more features associated with the subsequent query;selecting, using the predicted relevance, the particular item ofcontent; and providing data identifying the particular item of contentin response to the subsequent query.
 2. The method of claim 1, whereinthe particular item is a video, document, advertisement, image, or anaudio file.
 3. The method of claim 1, in response to receiving thesubsequent query, further comprising: selecting, by the search system, asubset of search results that each identify a respective item of contentfrom items of content, the subset of search results including aparticular search result identifying the particular item of content; andranking, by the search system, the subset of search results based onpredicted relevancies for the items of content identified by the searchresults.
 4. The method of claim 1, wherein at least the one or morefeatures of the subsequent query comprises a location of a device thattransmitted the subsequent query, a language of the subsequent query,and a time of day.
 5. The method of claim 1, wherein each search resultis indexed in the search index by a token, and the token comprises adescription of a feature of the particular item associated with thepredicted relevance of selecting the particular item.
 6. The method ofclaim 1, further comprising: training, by the search system, thepredictive model using training data that associates a selection of aparticular item of content with one or more corresponding features. 7.The method of claim 1, in response to receiving the subsequent query,further comprising: determining a context of the subsequent query basedon characteristics of the selection of the particular item of content,wherein the characteristics are related to an identity of the particularitem of content or a topic of the particular item of content.
 8. Themethod of claim 1, wherein the predicted relevance is a likelihood thatthe particular item of content will be selected by a user in response tosubmitting a search query associated with the one or more features.
 9. Asystem comprising: one or more computers and one or more storage devicesstoring instructions that are operable, when executed by the one or morecomputers, to cause the one or more computers to perform operationscomprising: before receiving, at a search system, a subsequent querythat is associated with at least one or more features: generating, usinga predictive model and by the search system, data indicating a predictedrelevance of a particular item of content to a query that is associatedwith the one or more features, wherein the predicted relevance isgenerated by the predictive model based at least on the one or morefeatures; and storing, by the search system and in a record entry of adatabase for a search index, data that associates the particular item ofcontent and the one or more features with the data indicating thepredicted relevance generated by the predictive model; receiving, at thesearch system, the subsequent query that is associated with at least theone or more features; in response to receiving the subsequent query:accessing, by the search system, the record entry of the database forthe search index to identify the particular item and the predictedrelevance that was previously generated by the predictive model based atleast on the one or more features associated with the subsequent query;selecting, using the predicted relevance, the particular item ofcontent; and providing data identifying the particular item of contentin response to the subsequent query.
 10. The system of claim 9, whereinthe particular item is a video, document, advertisement, image, or anaudio file.
 11. The system of claim 9, in response to receiving thesubsequent query, further comprising: selecting, by the search system, asubset of search results that each identify a respective item of contentfrom items of content, the subset of search results including aparticular search result identifying the particular item of content; andranking, by the search system, the subset of search results based onpredicted relevancies for the items of content identified by the searchresults.
 12. The system of claim 9, wherein at least the one or morefeatures of the subsequent query comprises a location of a device thattransmitted the subsequent query, a language of the subsequent query,and a time of day.
 13. The system of claim 9, wherein each search resultis indexed in the search index by a token, and the token comprises adescription of a feature of the particular item associated with thepredicted relevance of selecting the particular item.
 14. The system ofclaim 9, further comprising: training, by the search system, thepredictive model using training data that associates a selection of aparticular item of content with one or more corresponding features. 15.The system of claim 9, in response to receiving the subsequent query,further comprising: determining a context of the subsequent query basedon characteristics of the selection of the particular item of content,wherein the characteristics are related to an identity of the particularitem of content or a topic of the particular item of content.
 16. Thesystem of claim 9, wherein the predicted relevance is a likelihood thatthe particular item of content will be selected by a user in response tosubmitting a search query associated with the one or more features. 17.A non-transitory computer-readable medium storing software comprisinginstructions executable by one or more computers which, upon suchexecution, cause the one or more computers to perform operationscomprising: before receiving, at a search system, a subsequent querythat is associated with at least one or more features: generating, usinga predictive model and by the search system, data indicating a predictedrelevance of a particular item of content to a query that is associatedwith the one or more features, wherein the predicted relevance isgenerated by the predictive model based at least on the one or morefeatures; and storing, by the search system and in a record entry of adatabase for a search index, data that associates the particular item ofcontent and the one or more features with the data indicating thepredicted relevance generated by the predictive model; receiving, at thesearch system, the subsequent query that is associated with at least theone or more features; in response to receiving the subsequent query:accessing, by the search system, the record entry of the database forthe search index to identify the particular item and the predictedrelevance that was previously generated by the predictive model based atleast on the one or more features associated with the subsequent query;selecting, using the predicted relevance, the particular item ofcontent; and providing data identifying the particular item of contentin response to the subsequent query.
 18. The non-transitorycomputer-readable medium of claim 17, the particular item is a video,document, advertisement, image, or an audio file.
 19. The non-transitorycomputer-readable medium of claim 17, in response to receiving thesubsequent query, further comprising: selecting, by the search system, asubset of search results that each identify a respective item of contentfrom items of content, the subset of search results including aparticular search result identifying the particular item of content; andranking, by the search system, the subset of search results based onpredicted relevancies for the items of content identified by the searchresults.
 20. The non-transitory computer-readable medium of claim 17,wherein at least the one or more features of the subsequent querycomprises a location of a device that transmitted the subsequent query,a language of the subsequent query, and a time of day.